Image processing apparatus, image processing method, and non-transitory computer readable medium

ABSTRACT

An image processing apparatus includes a specifying unit, an extraction unit, and a creation unit. The specifying unit specifies, for one image among a first image before color conversion and a second image after color conversion, an area for which image information is extracted. The extraction unit extracts plural pieces of color conversion information, which are image information about pixels in the area of the one image among the first image and the second image, the area being specified by the specifying unit, and image information about pixels in the other image corresponding to the pixels in the one image. The creation unit creates a color conversion property on the basis of the plural pieces of color conversion information extracted by the extraction unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2017-120823 filed Jun. 20, 2017 andJapanese Patent Application No. 2017-220878 filed Nov. 16, 2017.

BACKGROUND (i) Technical Field

The present invention relates to an image processing apparatus, an imageprocessing method, and a non-transitory computer readable medium.

(ii) Related Art

With the widespread use of digital cameras, smartphones, tablets, and soon, the number of users who capture and view digital images is currentlyincreasing. Such images are captured in various environments that areaffected by illumination light and so on. Further, images of varioussubjects are captured. Therefore, after image capturing, a capturedimage may be found to be an image not expected by the user, and the useroften adjusts, for example, the color tone of the captured image.

SUMMARY

According to an aspect of the invention, there is provided an imageprocessing apparatus including a specifying unit, an extraction unit,and a creation unit. The specifying unit specifies, for one image amonga first image before color conversion and a second image after colorconversion, an area for which image information is extracted. Theextraction unit extracts plural pieces of color conversion information,which are image information about pixels in the area of the one imageamong the first image and the second image, the area being specified bythe specifying unit, and image information about pixels in the otherimage corresponding to the pixels in the one image. The creation unitcreates a color conversion property on the basis of the plural pieces ofcolor conversion information extracted by the extraction unit.

BRIEF DESCRIPTION OF THE DRAWINGS

An exemplary embodiment of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a diagram illustrating an example configuration of an imageprocessing system according to an exemplary embodiment;

FIG. 2 is a block diagram illustrating an example functionalconfiguration of an image processing apparatus according to an exemplaryembodiment;

FIG. 3 is a diagram illustrating examples of image data obtained by animage data obtaining unit;

FIG. 4A illustrates a case of determining, on the basis of a patternincluded in an area, position intervals at which color data isextracted, FIG. 4B illustrates a case of determining, on the basis ofthe number of colors and/or the number of tones, position intervals atwhich color data is extracted, and FIG. 4C illustrates a case ofdetermining, on the basis of the number of colors included in boxes aswell as the number of colors and/or the number of tones, positionintervals at which color data is extracted;

FIGS. 5A and 5B are diagrams illustrating example pairs of first colordata and second color data;

FIG. 6 illustrates a first example screen for checking color dataextracted by a color data extraction unit;

FIG. 7 illustrates a second example screen for checking color dataextracted by the color data extraction unit;

FIG. 8 is a diagram illustrating an example color conversion model;

FIGS. 9A and 9B are diagrams for comparing a case of creating a colorconversion model so that the relationship between first color data andsecond color data is a monotone increasing function with a case ofcreating a color conversion model so that the relationship is not amonotone increasing function;

FIG. 10 is a flowchart for describing an operation of the imageprocessing apparatus; and

FIG. 11 is a diagram illustrating a hardware configuration of the imageprocessing apparatus.

DETAILED DESCRIPTION

Hereinafter, an exemplary embodiment of the present invention will bedescribed in detail with reference to the attached drawings.

Description of Image Processing System

FIG. 1 is a diagram illustrating an example configuration of an imageprocessing system 1 according to this exemplary embodiment.

As illustrated in FIG. 1, the image processing system 1 according tothis exemplary embodiment includes an image processing apparatus 10, adisplay device 20, an input device 30, and a camera 40. The imageprocessing apparatus 10 makes a color adjustment (color conversion) toan original image captured by using the camera 40. The display device 20displays an image on the basis of image data output from the imageprocessing apparatus 10. The input device 30 is used by a user to inputvarious types of information to the image processing apparatus 10. Thecamera 40 is used to capture an image of an image-capture subject S andgenerates image data to which a color adjustment is made by the imageprocessing apparatus 10.

The image processing apparatus 10 is, for example, a general-purposepersonal computer (PC). The image processing apparatus 10 runs varioustypes of application software under the control of an operating system(OS) to thereby make a color adjustment and so on.

The display device 20 displays images on a display screen 21. Thedisplay device 20 is constituted by, for example, a liquid crystaldisplay for a PC, a liquid crystal display television, or a projectorthat has a function of displaying images by additive color mixture.Therefore, the display system of the display device 20 is not limited toa liquid crystal display system. In the example illustrated in FIG. 1,the display screen 21 is provided on the display device 20. In a case ofusing, for example, a projector as the display device 20, the displayscreen 21 is a screen or the like that is provided external to thedisplay device 20.

The input device 30 is constituted by a keyboard, a mouse, and so on.The input device 30 is used to input instructions for activating andterminating application software for a color adjustment and instructionsgiven by a user to the image processing apparatus 10 for making a coloradjustment in a case of making a color adjustment, which will bedescribed in detail below.

The camera 40 is an example of an image capturing apparatus andincludes, for example, an optical system that converges incident lightand an image sensor that is an image sensing unit detecting the lightconverged by the optical system.

The optical system is formed of a single lens or is formed by combiningplural lenses. In the optical system, for example, lenses are combinedand the surfaces of the lenses are coated to thereby remove variousaberrations. The image sensor is formed by arranging image sensingdevices, such as charge-coupled devices (CCDs) or complementarymetal-oxide semiconductors (CMOSs).

The image processing apparatus 10 and the display device 20 areconnected to each other via Digital Visual Interface (DVI) and may beconnected to each other via, for example, High-Definition MultimediaInterface (HDMI) (registered trademark) or DisplayPort instead of DVI.

The image processing apparatus 10 and the input device 30 are connectedto each other via Universal Serial Bus (USB) and may be connected toeach other via, for example, IEEE 1394 or RS-232C instead of USB.

The image processing apparatus 10 and the camera 40 are connected toeach other via a wireline in the example illustrated in FIG. 1 and areconnected to each other via, for example, USB, IEEE 1394, or RS-232C.Accordingly, image data of an image captured by using the camera 40 istransmitted to the image processing apparatus 10 via the wireline.However, the connection is not limited to this, and a wirelessconnection, such as a wireless local area network (LAN) or Bluetooth(registered trademark), may be used. The image processing apparatus 10and the camera 40 need not be connected to each other, and the camera 40may pass image data to the image processing apparatus 10 via, forexample, a memory card, such as an SD card.

In the image processing system 1 thus configured, first, a user capturesan image of the image-capture subject S by using the camera 40. Theimage captured by using the camera 40 is an original image, which is afirst image, and data of this image is transmitted to the imageprocessing apparatus 10. On the display device 20, the original image,which is an image before color processing, is displayed. Next, when theuser uses the input device 30 to input an instruction given to the imageprocessing apparatus 10 for making a color adjustment, the imageprocessing apparatus 10 makes a color adjustment to the original image.The result of this color adjustment is reflected to, for example, theimage displayed on the display device 20, and an image after the coloradjustment, which is a second image different from the first image, isdrawn and displayed on the display device 20. In this case, the user isable to make a color adjustment interactively while viewing the imagedisplayed on the display device 20, and therefore, is able to perform acolor adjustment operation more intuitively and more easily.

The second image need not be the image after a color adjustment made bythe image processing apparatus 10 and may be an image captured by usinganother camera having properties, that is, image-capture conditions,different from those of the camera 40. In this case, an image capturedby using the camera 40 is regarded as the first image, and an imagecaptured by using a camera having different image-capture conditions isregarded as the second image.

On the basis of the result of the color adjustment, the image processingapparatus 10 creates a conversion relationship that is used to make acolor adjustment to an original image and to obtain an image after thecolor adjustment. When it is assumed that, for example, first and secondcolor data is RGB data composed of red (R), green (G), and blue (B)data, and that the first color data is represented by (R_(a), G_(a),B_(a)) and the second color data is represented by (R_(b), G_(b),B_(b)), the color adjustment is a process of (R_(a), G_(a),B_(a))→(R_(b), G_(b), B_(b)). This conversion relationship is used toreproduce a color adjustment similar to the color adjustment previouslymade. The conversion relationship is also called a profile and iscreated as a 3D LUT. However, the conversion relationship is not limitedto this. For example, the conversion relationship may be created as aone-dimensional (1D) LUT representing R_(a)→R_(b), G_(a)→G_(b), andB_(a)→B_(b). The conversion relationship may be created as amultidimensional matrix representing (R_(a), G_(a), B_(a))→(R_(b),G_(b), B_(b)). The process for creating the conversion relationship willbe described below.

The image processing system 1 according to this exemplary embodiment isnot limited to the form illustrated in FIG. 1. The image processingsystem 1 may be, for example, a tablet terminal. In this case, thetablet terminal includes a touch panel, and the touch panel displaysimages and is used to input user instructions by, for example, touching.That is, the touch panel functions as the display device 20 and theinput device 30. As the camera 40, a camera built in the tablet terminalmay be used. As an apparatus into which the display device 20 and theinput device 30 are integrated, a touch monitor may be used. The touchmonitor includes a touch panel that is used as the display screen 21 ofthe display device 20. In this case, an image is displayed on the touchmonitor on the basis of image data output from the image processingapparatus 10. The user inputs an instruction for making a coloradjustment by, for example, touching the touch monitor.

In order for the image processing apparatus 10 to create a conversionrelationship, the image processing apparatus 10 needs to obtain firstcolor data from an original image, which is an image before a coloradjustment (color conversion), and to obtain second color data from animage after a color adjustment (color conversion), as described above.

At this time, if the number of portions in an image from which colordata is obtained is too small, and the number of obtained pieces ofcolor data is too small accordingly, color data for a necessary tonevalue may be omitted, and the accuracy of the conversion relationshipmay decrease. If the number of portions in an image from which colordata is obtained is too large, and the number of obtained pieces ofcolor data is too large accordingly, for example, color data may beobtained from a portion that is not necessary for color conversion. As aconsequence, noise may be included in a color conversion model describedbelow. That is, color data for which the direction for retouchingdiffers may be included. As a result, a highly accurate color conversionmodel is unlikely to be created, and the accuracy of the conversionrelationship may decrease.

Accordingly, in this exemplary embodiment, the image processingapparatus 10 is configured as follows so that, when the image processingapparatus 10 creates a conversion relationship, the above-describedissue is unlikely to arise.

Description of Image Processing Apparatus

Now, the image processing apparatus 10 is described.

FIG. 2 is a block diagram illustrating an example functionalconfiguration of the image processing apparatus 10 according to thisexemplary embodiment. FIG. 2 illustrates functions related to thisexemplary embodiment selected from among various functions provided bythe image processing apparatus 10.

As illustrated in FIG. 2, the image processing apparatus 10 according tothis exemplary embodiment includes an image data obtaining unit 11, anarea determination unit 12, an extraction determination unit 13, a colordata extraction unit 14, an extraction result check unit 15, a colorconversion model creation unit 16, a conversion relationship creationunit 17, and an output unit 18. The image data obtaining unit 11 obtainsimage data. The area determination unit 12 determines an area for whichcolor data is extracted from image data. The extraction determinationunit 13 determines a method for extracting color data. The color dataextraction unit 14 extracts color data from image data. The extractionresult check unit 15 instructs a user to check the result of extractionof color data. The color conversion model creation unit 16 creates acolor conversion model. The conversion relationship creation unit 17creates a conversion relationship.

The image data obtaining unit 11 obtains first image data, which isimage data of an original image before a color adjustment (colorconversion), and second image data, which is image data after a coloradjustment (color conversion). The first and second image data is in adata format for display on the display device 20 and is, for example,RGB data described above. The image data obtaining unit 11 may obtainimage data in another data format and covert the image data to obtainRGB data.

FIG. 3 is a diagram illustrating examples of image data obtained by theimage data obtaining unit 11.

FIG. 3 illustrates a case where three sets of image data obtained bycapturing images of clothes that are items for sale and a person puttingon clothes that are items for sale are provided. In FIG. 3, (a) and (b)respectively illustrate first image data (image data before a coloradjustment) and second image data (image data after a color adjustment)when an image of a down jacket is captured. In FIG. 3, (c) and (d)respectively illustrate first image data and second image data when animage of a person putting on a shirt and pants is captured. In FIG. 3,(e) and (f) respectively illustrate first image data and second imagedata when an image of a dress is captured.

As described above, plural sets of first image data and second imagedata are provided to thereby obtain image data that includes a largernumber of colors.

The area determination unit 12 is an example of a specifying unit andspecifies, for one of an original image before a color adjustment(before color conversion) and an image after a color adjustment (aftercolor conversion), an area for which color data (image information) isextracted.

That is, the area determination unit 12 determines positions at whichcolor data is obtained from the image data illustrated in FIG. 3. Acolor adjustment is made to a portion corresponding to clothes that areitems for sale. That is, it is requested that the color of an item forsale is reproduced more precisely so that the color of the actual itemfor sale matches the color of the item for sale displayed as an image.Therefore, an item for sale is likely to be a target of a coloradjustment. On the other hand, such a request is unlikely to be made tothe area of the background of an item for sale, and the background isunlikely to be a target of a color adjustment. Therefore, the areadetermination unit 12 determines the portion corresponding to clothes tobe an area for which color data is extracted.

Specifically, the area determination unit 12 determines a portion otherthan the background to be an area for which color data is extracted.Accordingly, the area determination unit 12 needs to determine thebackground and a portion other than the background. Image data of thebackground is substantially the same as image data of a left end portionof the image. Therefore, a portion in which image data significantlydiffers from the image data of a left end portion of the image isdetermined to be a portion other than the background. In order to sampleimage data that is compared with the image data of a left end portion ofthe image, for example, pixel positions are selected at predeterminedintervals in the image, and image data of a pixel at each pixel positionis compared with image data of a pixel in a left end portion of theimage. Alternatively, a mask having a predetermined size may be appliedto the image data, and the average value of image data within the maskmay be compared with the value of a pixel in a left end portion of theimage.

As another method for determining the area, the area determination unit12 performs a frequency analysis on the basis of the image data andobtains a pixel position at which a high frequency is produced. Thispixel position corresponds to the outline of a portion other than thebackground, and therefore, the area determination unit 12 determines theportion inside the outline to be a portion other than the background.Further, as another method for determining the area, the areadetermination unit 12 defines in advance an area centered on the centerof the image and having a predetermined size and determines a portionwithin the area to be a portion other than the background.

It is preferable the area determination unit 12 determine an area forwhich color data is extracted by further excluding a portion having askin color of a person. That is, it is desirable not to make a coloradjustment to a skin color of a person. A color adjustment to a skincolor of a person may result in an unnatural color. Therefore, it isdesirable not to include a portion having a skin color of a person as anarea for which color data is extracted.

The area determination unit 12 determines an area for which color datais extracted for each set of first image data and second image dataobtained by the image data obtaining unit 11. In the examplesillustrated in FIG. 3, the area determination unit 12 determines, foreach of the three sets of first image data and second image data, anarea for which color data is extracted by using the first image data andthe second image data. However, the area determination unit 12 need notuse all sets of first image data and second image data and may use someof the sets of first image data and second image data. In a case ofusing some of the sets of first image data and second image data, thearea determination unit 12 selects sets at random, for example. In acase where a large number of pieces of image data are obtained, the areadetermination unit 12 may perform a process for selecting pieces ofimage data as described above.

The extraction determination unit 13 is an example of a determinationunit and determines intervals at which color data is extracted in anarea specified by the area determination unit 12. Therefore, as colordata is extracted at shorter intervals, the space between positions atwhich color data is extracted in the area decreases, and the number ofpieces of color data that are extracted increases. On the other hand, ascolor data is extracted at longer intervals, the space between positionsat which color data is extracted in the area increases, and the numberof pieces of color data that are extracted decreases. If the intervalsat which color data is extracted remain unchanged, the number of piecesof color data that are extracted decreases as the number of pixels inthe area decreases, and the number of pieces of color data that areextracted increases as the number of pixels in the area increases.

Specifically, the extraction determination unit 13 determines intervalsat which color data is extracted on the basis of at least one of thenumber of colors, the number of tones, and a pattern included in thearea.

FIG. 4A illustrates a case of determining, on the basis of a patternincluded in the area, intervals at which color data is extracted.

Here, the extraction determination unit 13 determines a pattern byperforming a frequency analysis. In FIG. 4A, the horizontal axisrepresents the frequency in the area, and the vertical axis representsthe number of pieces of data from which color data is extracted.

In this case, as the frequency decreases, intervals at which color datais extracted are made longer, and as the frequency increases, intervalsat which color data is extracted are made shorter. That is, in a casewhere a pattern of, for example, clothes is not fine, intervals at whichcolor data is extracted are made longer to decrease the number of piecesof color data that are extracted. On the other hand, in a case where apattern of, for example, clothes is fine, intervals at which color datais extracted are made shorter to increase the number of pieces of colordata that are extracted.

FIG. 4B illustrates a case of determining, on the basis of the number ofcolors and/or the number of tones, intervals at which color data isextracted.

Here, the extraction determination unit 13 assumes that color areas(boxes) are obtained by dividing a color space with a predeterminedmethod and determines intervals at which color data is extracted on thebasis of the number of color areas (the number of boxes) that include acolor included in the area determined by the area determination unit 12.As the number of colors and/or the number of tones included in the areadetermined by the area determination unit 12 increase, the number ofboxes that include a color included in the area increases, and as thenumber of colors and/or the number of tones included in the areadetermined by the area determination unit 12 decrease, the number ofboxes that include a color included in the area decreases. In FIG. 4B,the horizontal axis represents the number of boxes that include a colorincluded in the area determined by the area determination unit 12, andthe vertical axis represents the number of pieces of data from whichcolor data is extracted.

As illustrated in FIG. 4B, as the number of boxes that include a colorincluded in the area decreases, intervals at which color data isextracted are made longer to thereby decrease the number of pieces ofcolor data that are extracted, and as the number of boxes that include acolor included in the area increases, intervals at which color data isextracted are made shorter to thereby increase the number of pieces ofcolor data that are extracted. That is, in a case where the number ofcolors and/or the number of tones included in the area are small,intervals at which color data is extracted is made longer to therebydecrease the number of pieces of color data that are extracted. On theother hand, in a case where the number of colors and/or the number oftones included in the area are large, intervals at which color data isextracted is made shorter to thereby increase the number of pieces ofcolor data that are extracted.

The extraction determination unit 13 may determine intervals at whichcolor data is extracted further on the basis of the number of colorsincluded in the boxes that include a color included in the area.

FIG. 4C illustrates a case of determining, on the basis of the number ofcolors included in the boxes that include a color included in the areaas well as the number of colors and/or the number of tones, intervals atwhich color data is extracted. In FIG. 4C, the horizontal axisrepresents the number of boxes that include at least a predeterminednumber of colors among the boxes that include a color included in thearea, and the vertical axis represents the number of pieces of data fromwhich color data is extracted.

As illustrated in FIG. 4C, as the number of boxes for which the numberof included colors is equal to or larger than a threshold decreases,intervals at which color data is extracted is made longer to therebydecrease the number of pieces of color data that are extracted, and asthe number of boxes for which the number of included colors is equal toor larger than the threshold increases, intervals at which color data isextracted is made shorter to thereby increase the number of pieces ofcolor data that are extracted. In this case, a box that includes only asmall number of colors is not counted. Also in this case, as the numberof colors and/or the number of tones included in the area decrease,intervals at which color data is extracted is made longer to therebydecrease the number of pieces of color data that are extracted. On theother hand, as the number of colors and/or the number of tones includedin the area increase, intervals at which color data is extracted is madeshorter to thereby increase the number of pieces of color data that areextracted.

The method for determining intervals at which color data is extracted isnot limited to the above-described method.

For example, the extraction determination unit 13 may determineintervals at which color data is extracted on the basis of the number ofpatterned images included in the area for which color data is extracted.In this case, as the number of patterned images included in the areadecreases, intervals at which color data is extracted is made longer,and as the number of patterned images included in the area increases,intervals at which color data is extracted is made shorter.

The extraction determination unit 13 may determine a weight that isadded to extracted color data in addition to intervals at which colordata is extracted or instead of intervals at which color data isextracted. That is, both intervals at which color data is extracted anda weight that is added to extracted color data may be used.Alternatively, intervals at which color data is extracted are madeconstant, and a weight that is added to extracted color data may beused.

The color data extraction unit 14 extracts, as color conversioninformation, color data from pixels in an area of one image among theoriginal image (first image) and the image after a color adjustment(second image), the area being determined by the area determination unit12, and from pixels in the other image corresponding to the pixels inthe one image at intervals for extracting color data determined by theextraction determination unit 13. In other words, the color dataextraction unit 14 extracts, as a set of pieces of color data atcorresponding positions in the images, first color data from the firstimage data and second color data from the second image data. In thiscase, the set of the extracted first color data and the extracted secondcolor data is color conversion information.

That is, the color data extraction unit 14 extracts, from the firstimage data and the second image data, first color data and second colordata at the same positions in the images.

FIGS. 5A and 5B are diagrams illustrating example pairs of first colordata and second color data.

FIG. 5A illustrates an example original image that is an image before acolor adjustment and example pieces of first color data. In FIG. 5A, theoriginal image is an image of a blouse, from which pieces of color dataare extracted at portions indicated by 1 to 5 and are respectivelyindicated as RGBa1 to RGBa5. In this case, the blouse is entirely blue,and therefore, all of the pieces of color data RGBa1 to RGBa5 are piecesof RGB data indicating blue.

FIG. 5B illustrates an example image after a color adjustment andexample pieces of second color data. In FIG. 5B, pieces of color datarespectively extracted from the same portions indicated by 1 to 5 as inFIG. 5A are indicated as RGBb1 to RGBb5.

The extraction result check unit 15 is an example of an extractionresult check unit and outputs color data (color conversion information)extracted by the color data extraction unit 14 for display on thedisplay device 20.

FIG. 6 illustrates a first example screen for checking color dataextracted by the color data extraction unit 14.

FIG. 6 illustrates a case where the distribution of the color dataextracted by the color data extraction unit 14 is displayed as“distribution of obtained color data”, and the number of pieces of colordata that have been extracted is indicated on a scale of 1 to 4 for eachof the twelve colors, namely, white through pink. This exampleillustrates a case where a necessary number of pieces of color data havebeen obtained for, for example, white, black, gray, blue, purple, andpink, but a necessary number of pieces of color data have not beenobtained for the other colors, particularly, yellow, green, and lightblue. Accordingly, a notification “The number of images for learning istoo small. Please increase the number of images that are set.” is sentto the user as a result of diagnosis. That is, the number of sets offirst color data and second color data obtained by the image dataobtaining unit 11 is insufficient, and the number of pieces of colordata extracted by the color data extraction unit 14 is not sufficientaccordingly. The extraction result check unit 15 is able to determinewhether the number of pieces of color data that have been extracted issufficient or not.

FIG. 7 illustrates a second example screen for checking color dataextracted by the color data extraction unit 14.

FIG. 7 illustrates a case where the distribution of the color dataextracted by the color data extraction unit 14 is plotted in an RGBcolor space and displayed on the display device 20 as a color spaceimage. In FIG. 7, the black dots represent pieces of color dataextracted by the color data extraction unit 14.

Accordingly, the user is able to visually know the distribution of theextracted pieces of color data. At this time, the extraction resultcheck unit 15 is able to delete some of the extracted pieces of firstcolor data and second color data (color conversion information) inaccordance with an instruction from the user. This operation isperformed by, for example, the user deleting a piece of color datadetermined not to be necessary among the pieces of color dataillustrated in FIG. 7.

The color conversion model creation unit 16 is an example of a creationunit and creates a color conversion property (color conversion model) onthe basis of plural sets of first color data and second color data(color conversion information) extracted by the color data extractionunit 14. That is, the color conversion model creation unit 16 creates acolor conversion model that represents the relationship between thefirst color data and the second color data extracted by the color dataextraction unit 14.

FIG. 8 is a diagram illustrating an example color conversion model.

In FIG. 8, the horizontal axis represents first color data, which iscolor data before a color adjustment, and the vertical axis representssecond color data, which is color data after a color adjustment. Thefirst and second color data is RGB data. In FIG. 8, the first color datais indicated as RGBa, and the second color data is indicated as RGBb.

The black dots represent the result of plotting the pieces of firstcolor data and the pieces of second color data extracted by the colordata extraction unit 14. FIG. 8 illustrates a case where twelve pairs offirst color data and second color data have been extracted by the colordata extraction unit 14.

The solid line represents the relationship between the first color dataand the second color data and represents the color conversion modelcreated by the color conversion model creation unit 16. That is, thecolor conversion model may be regarded as a function that represents therelationship between the first color data and the second color data.When this function is expressed by f, the color conversion model isexpressed by RGBb=f(RGBa). The color conversion model may be created byusing a publicly known method. However, it is preferable that a methodhaving high fitting performance for nonlinear characteristics, namely, aweighted regression model or a neural network, be used. Note thatnonlinear characteristics need not be used, and linear characteristicsusing a matrix model may be used.

It is preferable that the color conversion model creation unit 16 createa color conversion model so that the relationship between the firstcolor data and the second color data is a nonlinear monotone increasingfunction.

FIGS. 9A and 9B are diagrams for comparing a case of creating a colorconversion model so that the relationship between first color data andsecond color data is a monotone increasing function with a case ofcreating a color conversion model so that the relationship is not amonotone increasing function.

In each of FIGS. 9A and 9B, the curves represented by solid linesrepresent color conversion models. Among the color conversion models, acolor conversion model represented by a thick line is created so thatthe relationship between the first color data and the second color datais a monotone increasing function, and is a color conversion modelsimilar to that in FIG. 8. Among the color conversion models, a colorconversion model represented by a thin line is created so that therelationship between the first color data and the second color data isnot a monotone increasing function. The monotone increasing functiondescribed here is a function with which the slope of the tangent linetouching the solid line that represents the color conversion model isequal to or larger than 0° at any point, and the slope may be equal to0° at some points. That is, the monotone increasing function describedhere is a monotone increasing function in a broad sense.

In the color conversion model represented by a thick line, the slope ofthe tangent line is not smaller than 0° (not negative) at any point andis equal to or larger than 0° at all points.

On the other hand, in the color conversion model represented by a thinline, the slope of the tangent line is smaller than 0° (negative) atsome points. That is, as RGBa increases, RGBb decreases in someportions. When such a color conversion model is used to create aconversion relationship, and the conversion relationship thus created isused to make a color adjustment, an image after the color adjustment mayinclude a tone jump. When a color conversion model is created so thatthe relationship between the first color data and the second color datais a monotone increasing function, the possibility of a tone jump isreduced, and furthermore, variations in a color adjustment are reduced.

In order for the color conversion model creation unit 16 to create acolor conversion model as represented by a thick line, it is preferablethat the number of pairs of first color data and second color data likethe pairs indicated by Pr1, Pr2, and Pr3 be smaller. The pairs of firstcolor data and second color data indicated by Pr1, Pr2, and Pr3 arecolor data for which the direction for retouching differs, as describedabove. In this exemplary embodiment, the image data obtaining unit 11obtains pieces of first image data, which are image data before a coloradjustment, and pieces of second image data, which are image data aftera color adjustment, and from these pieces of image data, pieces of colordata for various colors are obtained as sets of first color data andsecond color data to thereby reduce the possibility of color data forwhich the direction for retouching differs being included. Theextraction determination unit 13 determines intervals at which colordata is extracted, and the color data extraction unit 14 extracts anappropriate number of pieces of color data to thereby further reduce thepossibility of color data for which the direction for retouching differsbeing included. As described with reference to FIG. 7, the user mayperform an operation for deleting a piece of color data determined notto be necessary to thereby further reduce the possibility of color datafor which the direction for retouching differs being included.

In a case where a weight is set for a pair of first color data andsecond color data, the color conversion model creation unit 16 creates acolor conversion model by using the weight.

The conversion relationship creation unit 17 creates a conversionrelationship that is used to reproduce a color adjustment made to thefirst image (original image) on the basis of the color conversion modelcreated by the color conversion model creation unit 16. The colorconversion model is created from sets of first color data and secondcolor data (color conversion information), and therefore, in otherwords, the conversion relationship creation unit 17 creates a conversionrelationship that is used to reproduce a color adjustment on the basisof the sets of first color data and second color data. The conversionrelationship is used to reproduce the result of a color adjustment madeto an original image by the user. That is, when a color adjustment ismade to an image (original image) before a color adjustment by using theconversion relationship, a color adjustment similar to a coloradjustment previously made by the user is made again to obtain an imageafter the color adjustment.

In a case where the conversion relationship is created as a 3D LUT, arepresentative pixel value is selected for each of R, G, and B. Forexample, in a case where data of each of R, G, and B is represented byan 8-bit tone value, the pixel value is an integer ranging from 0 to255. Here, the pixel value is divided into, for example, eight values.Then, RGB data represented by each of the pixel values obtained by thedivision into eight values is assumed to be a lattice point (ninelattice points). In this case, the number of lattice points is 9³=729.For each of these lattice points, the relationship between the firstcolor data and the second color data is calculated on the basis of thecolor conversion model. The relationship thus calculated is representedby an LUT, which is a 3D LUT. Accordingly, the 3D LUT is described as arelationship expressed by input value (R_(a), G_(a), B_(a))−output value(R_(b), G_(b), B_(b)) for each lattice point.

It is preferable that the conversion relationship creation unit 17create a conversion relationship that enables a color adjustment to acolor included in the area determined by the area determination unit 12and that does not enable a color adjustment to the other colors. Forexample, in the case of the images illustrated in FIGS. 5A and 5B, theconversion relationship creation unit 17 creates a conversionrelationship that enables a color adjustment to the blue color area andthat does not enable a color adjustment to the other colors. In the casewhere the conversion relationship is represented by a 3D LUT, the inputvalue (R_(a), G_(a), B_(a)) and the output value (R_(b), G_(b), B_(b))of a lattice point that corresponds to a color close to a color includedin the area determined by the area determination unit 12 are differentvalues, and the input value (R_(a), G_(a), B_(a)) and the output value(R_(b), G_(b), B_(b)) of any other lattice point are the same values inthe 3D LUT. Therefore, in a case where the area determination unit 12determines an area for which color data is extracted while furtherexcluding a portion having a skin color of a person, a color adjustmentis not made to a skin color of a person.

The output unit 18 is an example of a color adjustment unit that usesthe conversion relationship to make a color adjustment to an imagecaptured by using the camera 40, and outputs image data after the coloradjustment and data of the conversion relationship. The image data afterthe color adjustment is output to the display device 20, and the displaydevice 20 displays an image after the color adjustment on the basis ofthe image data. The data of the conversion relationship is stored on,for example, the image processing apparatus 10, and the conversionrelationship is used to make a color adjustment. The conversionrelationship may be output to an external device other than the imageprocessing apparatus 10, and the external device may use the conversionrelationship to make a color adjustment.

Now, an operation of the image processing apparatus 10 is described.

FIG. 10 is a flowchart for describing an operation of the imageprocessing apparatus 10. The operation of the image processing apparatus10 described below may be regarded as an image processing method that isused by the image processing apparatus 10.

The image data obtaining unit 11 obtains first image data, which isimage data of an original image before a color adjustment, and secondimage data, which is image data after a color adjustment (step S101:image data obtaining step).

Next, the area determination unit 12 specifies, for one of the originalimage before a color adjustment and the image after a color adjustment,an area for which color data is extracted (step S102: specifying step).At this time, the area determination unit 12 determines a portion otherthan the background to be an area for which color data is extracted. Itis preferable that the area determination unit 12 determine an area forwhich color data is extracted while further excluding a portion having askin color of a person.

Subsequently, the extraction determination unit 13 determines intervalsat which color data is extracted in the area specified by the areadetermination unit 12 (step S103: determination step). At this time, theextraction determination unit 13 determines intervals at which colordata is extracted on the basis of at least one of the number of colors,the number of tones, and a pattern included in the area.

Subsequently, the color data extraction unit 14 extracts, as colorconversion information, color data from pixels in the area of one imageamong the original image and the image after a color adjustment, thearea being specified by the area determination unit 12, and from pixelsin the other image corresponding to the pixels in the one image at theintervals for extracting color data determined by the extractiondetermination unit 13 (step S104: extraction step).

Subsequently, the extraction result check unit 15 outputs the color dataextracted by the color data extraction unit 14 for display on thedisplay device 20 (step S105, extraction result checking step).

At this time, the user may be allowed to perform an operation fordeleting color data that is determined not to be necessary, as describedwith reference to FIG. 7.

Subsequently, the color conversion model creation unit 16 creates acolor conversion model on the basis of plural sets of first color dataand second color data (color conversion information) extracted by thecolor data extraction unit 14 (step S106: creation step).

The conversion relationship creation unit 17 creates a conversionrelationship that is used to reproduce the color adjustment on the basisof the color conversion model created by the color conversion modelcreation unit 16 (step S107: conversion relationship creation step). Theconversion relationship is created as, for example, a 3D LUT asdescribed above. The conversion relationship may be output in a widelyknown format, such as the International Color Consortium (ICC) profileformat.

Thereafter, the output unit 18 outputs image data after a coloradjustment and data of the conversion relationship (step S108).

In this exemplary embodiment, a color conversion model is created whileexcluding color data for which the direction for retouching differs tothereby create a more ideal color conversion model, and furthermore,create a highly accurate conversion relationship. When such a colorconversion model is used to create output data (for example, an ICCprofile), and the output data is used to perform color conversion on animage captured by using the camera 40, a more ideal color adjustment ismade.

In the above-described example, the original image is an image capturedby using the camera 40; however, the original image is not specificallylimited. For example, the original image may be an image read by ascanner. Alternatively, commercially available image data or image datadistributed via, for example, the Internet may be used as is as theoriginal image.

Example Hardware Configuration of Image Processing Apparatus

Now, a hardware configuration of the image processing apparatus 10 isdescribed.

FIG. 11 is a diagram illustrating a hardware configuration of the imageprocessing apparatus 10.

The image processing apparatus 10 is implemented as, for example, a PCas described above. As illustrated in FIG. 11, the image processingapparatus 10 includes a central processing unit (CPU) 91, which is anarithmetic processing unit, a main memory 92, which is a memory, and ahard disk drive (HDD) 93. The CPU 91 executes various programs, such asan operating system (OS) and application software. The main memory 92 isa storage area for storing various programs, data used when the programsare executed, and so on. The HDD 93 is a storage area for storing inputdata to various programs, output data from various programs, and so on.

The image processing apparatus 10 further includes a communicationinterface (I/F) 94 for external communication.

Description of Program

The processing performed by the image processing apparatus 10 in theexemplary embodiment described above is provided as, for example aprogram, such as application software.

Therefore, the processing performed by the image processing apparatus 10in the exemplary embodiment may be regarded as a program for causing acomputer to implement a specifying function of specifying, for one imageamong a first image and a second image, an area for which color data isextracted; an extraction function of extracting plural pieces of colorconversion information, which are color data of pixels in the area ofthe one image among the first image and the second image, the area beingspecified by the specifying function, and color data of pixels in theother image corresponding to the pixels in the one image; and a creationfunction of creating a color conversion model on the basis of the pluralpieces of color conversion information extracted by the extractionfunction.

The program for implementing the exemplary embodiment may be providedvia a communication system, as a matter of course, or may be stored in arecording medium, such as a compact disc read-only memory (CD-ROM), andprovided.

The exemplary embodiment has been described; however, the technicalscope of the present invention is not limited to the scope of theabove-described exemplary embodiment. It is obvious from the descriptionof the claims that various modifications and alterations made to theabove-described exemplary embodiment are included in the technical scopeof the present invention.

The foregoing description of the exemplary embodiment of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. An image processing apparatus comprising: amemory; and a processor configured to execute: a specifying unitconfigured to specify, for one image among a first image before colorconversion and a second image after color conversion, an area for whichimage information is extracted; an extraction unit configured to extracta plurality of pieces of color conversion information, which are imageinformation about pixels in the area of the one image among the firstimage and the second image, the area being specified by the specifyingunit, and image information about pixels in the other imagecorresponding to the pixels in the one image; and a creation unitconfigured to create a color conversion property on the basis of theplurality of pieces of color conversion information extracted by theextraction unit, wherein the specifying unit is configured to specify aportion other than a background as the area for which image informationis extracted.
 2. The image processing apparatus according to claim 1,wherein the specifying unit is configured to specify the area for whichimage information is extracted while further excluding a portion havinga skin color of a person.
 3. The image processing apparatus according toclaim 1, wherein the first image and the second image are received froma user.
 4. The image processing apparatus according to claim 1, whereinthe processor is further configured to execute: an extraction resultcheck unit configured to determine whether a number of the plurality ofpieces of color conversion information extracted by the extraction unitis sufficient for creating the color conversion property.
 5. An imageprocessing apparatus comprising: a memory; and a processor configured toexecute: a specifying unit configured to specify, for one image among afirst image before color conversion and a second image after colorconversion, an area for which image information is extracted; anextraction unit configured to extract a plurality of pieces of colorconversion information, which are image information about pixels in thearea of the one image among the first image and the second image, thearea being specified by the specifying unit, and image information aboutpixels in the other image corresponding to the pixels in the one image;a creation unit configured to create a color conversion property on thebasis of the plurality of pieces of color conversion informationextracted by the extraction unit; and a determination unit configured todetermine intervals at which the image information is extracted in thearea specified by the specifying unit, wherein the extraction unit isconfigured to decide to extract the plurality of pieces of colorconversion information at the intervals for extracting the imageinformation determined by the determination unit.
 6. The imageprocessing apparatus according to claim 5, wherein the determinationunit is configured to determine the intervals at which the imageinformation is extracted on the basis of at least one of the number ofcolors, the number of tones, and a pattern included in the area.
 7. Theimage processing apparatus according to claim 6, wherein thedetermination unit is configured to determine the pattern by performinga frequency analysis.
 8. The image processing apparatus according toclaim 6, wherein the determination unit is configured to assume thatcolor areas are obtained by dividing a color space with a predeterminedmethod, and determine the intervals at which the image information isextracted on the basis of the number of the color areas that include acolor included in the area.
 9. The image processing apparatus accordingto claim 8, wherein the determination unit is configured to determinethe intervals at which the image information is extracted further on thebasis of the number of colors included in the color areas that include acolor included in the area.
 10. The image processing apparatus accordingto claim 5, wherein the determination unit is configured to determine aweight added to the plurality of extracted pieces of color conversioninformation in addition to the intervals at which the image informationis extracted or instead of the intervals at which the image informationis extracted.
 11. An image processing apparatus comprising: a memory;and a processor configured to execute: a specifying unit configured tospecify, for one image among a first image before color conversion and asecond image after color conversion, an area for which image informationis extracted; an extraction unit configured to extract a plurality ofpieces of color conversion information, which are image informationabout pixels in the area of the one image among the first image and thesecond image, the area being specified by the specifying unit, and imageinformation about pixels in the other image corresponding to the pixelsin the one image; a creation unit configured to create a colorconversion property on the basis of the plurality of pieces of colorconversion information extracted by the extraction unit; and anextraction result check unit configured to output the plurality ofpieces of color conversion information extracted by the extraction unitfor display on a display device.
 12. The image processing apparatusaccording to claim 11, wherein the extraction result check unit isconfigured to delete one or more of the plurality of extracted pieces ofcolor conversion information in accordance with a user instruction. 13.An image processing method comprising: specifying, for one image among afirst image before color conversion and a second image after colorconversion, an area for which image information is extracted; extractinga plurality of pieces of color conversion information, which are imageinformation about pixels in the area of the one image among the firstimage and the second image, the area being specified in the specifying,and image information about pixels in the other image corresponding tothe pixels in the one image; and creating a color conversion property onthe basis of the plurality of pieces of color conversion informationextracted in the extracting, wherein the area for which imageinformation is extracted includes a portion of the one image other thana background of the one image.
 14. A non-transitory computer readablemedium storing a program causing a computer to execute a process forimage processing, the process comprising: specifying, for one imageamong a first image before color conversion and a second image aftercolor conversion, an area for which image information is extracted;extracting a plurality of pieces of color conversion information, whichare image information about pixels in the area of the one image amongthe first image and the second image, the area being specified in thespecifying, and image information about pixels in the other imagecorresponding to the pixels in the one image; and creating a colorconversion property on the basis of the plurality of pieces of colorconversion information extracted in the extracting, wherein the area forwhich image information is extracted includes a portion of the one imageother than a background of the one image.